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(54) Systems and methods for generating and playback of annotated multimedia presentation 

(303) so that electronic mailers (530) or web browsers 
(510) can retrieve it, and enables the generation of a 
multimedia message, which consists of a collection of 
multimedia streams with graphical and synchronized 
audio/video annotations that can be mailed and re- 
trieved by electronic mailers. 
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a multimedia presentation using images, video, 3D 
graphics, digital fax, voice mail, HTML pages, and doc- 
uments with graphics and audio; converts the multime- 
dia presentation into a collection of multimedia streams 



CO 
CM 



Q. 

LU 




Printed by Jouve, 75001 PARIS (FR) 



1 



EP 1 126 721 A1 



Description 

BACKGROUND OF THE INVENTION 

FIELD OF THE INVENTION 5 

[0001] The present invention relates to multimedia 
communications and more particularly to using multime- 
dia streams to generate and playback presentations or 
messages that can consist of dynamic annotations on »? 
multiple images, videos, 3D graphics, digital fax and 
voice mail, and HTML and Microsoft Office documents. 

DISCRETION OF THE PRIOR ART 

15 

[0002] Due to a rapidly growing number of users and 
exponentially growing number of connections to the In- 
ternet, the Internet has become a widely accepted me- 
dium for communication, and as well as shopping and 
henceforth marketing. The ability to deliver multimedia 2 ° 
data over the Internet is limited by bandwidth capacity 
and cost of the network connection and by the compu- 
tational capability of the server and client computers. 
Recently; various companies have developed data 
streaming methods to deliver good quality multimedia 25 
data over even the low bandwidth capacity networks, 
such as 28.8 or 56 kbits modem connections. In addi- 
tion, the cost of the network connection has become af- 
fordable for many home users because of the fierce 
competition in ISP market. Furthermore, the computa- 30 
tional power of server and client computers is increasing 
exponentially, while their prices are going down rapidly. 
[0003] As a result of the above developments, hu- 
mans now can use the Internet heavily for communicat- 
ing with each other and sharing their information. How- 35 
ever, textual information sharing and communication is 
not always effective. A renaissance in WWW publishing 
has recently taken place. Many WWWsites contain mul- 
timedia streams, images, videos and 3D graphics to fa- 
cilitate and enhance the Internet based communication. 40 
To better express themselves, people are sending each 
other images, videos or 3D graphics in addition to their 
textual message. 

[0004] Until recently, the multimedia data on a WWW 
page or in a message are not linked semantically to each 45 
other but linked syntactically. However, people are mov- 
ing away rapidly from this framework. The development 
of new frameworks has been witnessed, where the au- 
dio ortalking head video annotations and other dynamic 
multimedia content graphically support the multimedia 50 
content and enhances the communication. For exam- 
ple, the streaming video is enriched by audio and textual 
annotations attached to a frame or instance in a video. 
This is described in "Annotations for Streaming Video 
on the Web: System Design and Usage Studies" by 55 
David Bargeron, Anoop Gupta, Jonathan Grudin, and 
Elizabeth Sanocki appeared in the Eight International 
World Wide Web Conference, Toronto Convention 



Center, Toronto Canada, May 11-14, 1 999. This system 
requires the video playback to be stopped to access the 
annotations. In another example, the power point pres- 
entations are supported by synchronized talking head 
video annotations [Eloquent Enterprise Communica- 
tions Portal, http://www.eloquent.com/]. This system is 
limited to only power point presentation enrichment. 
Hence, there is a need for developing messaging and 
communication systems and methods that allow seam- 
less integration of all other of multimedia content into a 
multimedia presentation for more effective communica- 

[0005] Another annotation concept, the overlays on 
the visual multimedia content (a.k.a. graphical annota- 
tions), is addressed for images in creating multimedia 
presentations in US Patent Number 5,83'B,31 3 entitled 
"Multimedia-based Reporting System With Recording 
And Playback Of Dynamic Annotation," by Ta-Yuan 
Hou, Wei-Kwan Vincent Su, Thomas Meissner and Ar- 
ding Hsu and assigned to the same assignee as the 
present invention. This system does not enable anno- 
tation of other multimedia contents such as video, 3D 
graphics, digital fax and voice mail, and HTML and Mi- 
crosoft Office documents such as Microsoft Word doc- 
uments, Excel spreadsheets and Power Point presen- 
tations. Another missing component in this system is the 
attachment of dynamic markers, animated drawings, 
and hyperlinks to regions or surfaces in the multimedia, 
content. For example, this system does not enable the 
generation of dynamic markers attached to moving vid- 
eo objects such as cars or people in a scene, animated 
(flashing) graphical drawings, or 3D markers that stick 
to a surface in the 3D scene, and scales or moves in 
that scene as the view point changes. In addition, this 
system does not allow HTML page annotation, thus link 
traversal. Therefore, there is a need for developing sys- 
tems and methods that enable the generation of seam- 
less integration of vast multimedia contents, i.e., multi- 
media content is not limited to images only, synchro- 
nized not only with audio/video annotations but also with 
graphical annotations like texts, dynamic freeform ani- 
mated or static graphical drawings, and dynamic region 
markers. In addition, such systems should allow dynam- 
ic hyperlink creation such as hotspots in a video se- 
quence, and hyperlink traversal such as in a web brows- 
er. 

[0006] Such media annotation systems will require 
proprietary plugins to playback the presentations within 
one of the conventional Internet applications, the web 
browser, since the file format for storing the seamlessly 
integrated (or composed) multimedia presentations is 
proprietary. Web browsers are now capable of receiving 
and presenting streaming multimedia content by the 
freely available plugins provided by RealNetworks Inc. 
(http://www.real.com/playerplus/) or Microsoft Corp. 
(http://www.microsoft.com/windows/mediaplayer/). 
Hence, the delivery of multimedia presentation over the 
Internet is no longer a problem if they are converted into 
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a collection of known and commonly supported multi- 
media streams. Therefore, the systems that enable the 
creation of multimedia presentations need methods and 
systems for generating multimedia streams of the pres- 
entations. 

[D007] Similarneeds exist when it is necessary to mail 
and playback a graphically annotated multimedia mes- 
sage. It is not always feasible to send a multimedia pres- 
entation as a message with attached content, because 
the multimedia message may be based on video or oth- 
er large size multimedia content. Therefore, it is also de- 
sirable to stream the multimedia content of the message 
during the playback. As in the case of publishing multi- 
media presentations on the WWW, it cannot be as- 
sumed that the electronic mail recipient would have the 
media annotation tool installed in his/her computer. 
Hence, it is desirable to send a self contained executa- 
ble as an attachment so that it displays the multimedia 
message. 

SUMMARY OF THE INVENTION 

[0008] The present Invention Is a system and method 
for: (i) generating and displaying a multimedia presen- 
tation using images, video, 3D graphics, digital fax, 
voice mail, HTML pages, and Microsoft Office docu- 
ments like Microsoft Word documents, Excel spread- 
sheets and Power Point presentations with graphical 
and audio, and/or talking head video annotations; (ii) 
converting the multimedia presentation into a collection 
of multimedia streams so that electronic mailers or web 
browsers can retrieve it either by the help of a freely 
available streaming media players such as RealPlayer 
by RealNetworks Inc. (http://www.real.com/playerplus/) 
or Windows Media Player by Microsoft Corp. 
(http://www.microsoft.corn/windows/mediaplayer/), 
plus if necessary a proprietary plugin to render the 
graphical annotations; and (iii) enabling the generation 
of a multimedia message, which consists of a collection 
of multimedia streams with graphical and synchronized 
audio/video annotations, that can be mailed and re- 
trieved by electronic mailers. This invention also pro- 
vides a system and method for eliminating the need for 
a proprietary plugin in the case where it is not desired 
by the clients. This latter system and method consist of 
generation of a slide show presentation of the visual 
content of the multimedia presentation throughout its 
duration and converting this slide show presentation into 
a multimedia stream. 

BRIEF DESCRIPTION OF THE DRAWINGS 

[0009] Additional objects and features of the invention 
will be more readily apparent from the following detailed 
description and appended claims when taken in con- 
junction with the drawings, in which: 
[0010] FIG, 1 is a block diagram of a preferred em- 
bodiment of the present invention for generating multi- 



media streams. 

[0011] FIG. 2 is a block diagram of a preferred em- 
bodiment of the present invention for generating multi- 
media presentations and retrieval of these presenta- 
s tions by a client computer. 

[0012] FIG. 3 is a block diagram of a preferred em- 
bodiment of the present invention for generating multi- 
media messages and retrieval of these messages by a 
client computer. 
10 [0013] FIG. 4 is a block diagram of a preferred em- 
bodiment of the present invention for generating multi- 
media messages with attachments and retrieval of 
these messages by a client computer. 
[0014] FIG. 5 is a block diagram of a preferred em- 
's bodiment of the plugin. 

[0015] FIG. 6 is a block diagram of a preferred em- 
bodiment of the media annotation tool. 
[0016] FIG. 7 is a block diagram of a preferred em- 
bodiment of the multimedia streams generator. 
20 [0017] FIG. 8 is a flow chart of a preferred embodi- 
ment of the multimedia presentation generation by the 
web publisher. 

[0018] FIG. 9 is a flow chart of a preferred embodi- 
ment of the multimedia message generation by the elec- 
ts tronic mailer. 

DETAILED DESCRIPTION OF THE INVENTION 

[0019] The present invention will now be described in 
30 detail with reference to a few preferred embodiments 
thereof as illustrated in the accompanying drawings. In 
the following description, numerous specific details are 
set forth in order to provide a thorough understanding 
of the present invention. It will be apparent, however, to 
35 one skilled in the art, that the present invention may be 
practiced without some or all of these specific details. In 
other instances, well known process steps have not 
been described in detail in order not to unnecessarily 
obscure the present invention. 
40 [0020] Referring to FIG. 1 , there is shown a block di- 
agram of a preferred embodiment for generating multi- 
media streams, which includes a Media Annotation Tool 
(MAT) 210 and a Multimedia Streams Generator (MSG) 
250. 

45 [0021] Information input to the media annotation tool 
as media sources 110 include digitized videos, images, 
3D graphics files, fax and voice mail, and HTML and Of- 
fice documents 111, digital camera output 112, digital 
video camera output 113, 3D scanner output 114, and 

so image scanner output 115. Information input to the me- 
dia annotation tool as annotation sources may include 
one or more of the live video capture outputs 1 21 , live 
audio capture outputs 122, computer mouse actions 
123, and computer keyboard entries 124. Information 

55 output by the media annotation tool 210 to the multime- 
dia streams generator 250 includes graphical annota- 
tions 301 with the audio and/or video annotations 302. 
Graphical annotations 301 include events associated to 
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the mouse actions and keyboard entries during the re- 
cording of a media annotation session. It is well known 
to those in the art that the mouse actions and keyboard 
entries can be caught by an application and interpreted 
according to specific rules defined by the system to gen- 
erate events. Each event has a time stamp that repre- 
sents its time of occurrence. An event could be: (i) to 
draw a shape with given coordinates and description on 
the display area of the input media, e.g., a polygonal 
marker that sticks to a video object in the video scene, 
or a 3D marker that sticks to a surface in the 3D scene, 
or a 3D volume that sticks to an object in the 3D anima- 
tion; (ii) to draw a static or animated line from the previ- 
ous position of the mouse to a new location on the dis- 
play area of the input media, e.g., showing the current 
flow in an electronic circuit; (iii) to erase such drawings; 
(iv) to place text that is centered at a position specified 
by the user on the display area of the input media; (iv) 
to erase such texts; or (v) a URL, document or presen- 
tation display command in a specified window in the cli- 
ent computer, e.g., supporting power point presenta- 
tions of a live distant learning lecture over the Internet. 
The audio annotations may include the narration of the 
semantics of graphical annotations, such as speech or 
background music. Finally, the video annotations may 
include the talking head image sequence associated 
with the audio annotations. 

[0022] Information input to the multimedia streams 
generator 250 includes the media sources 110, graphi- 
cal annotations 301 and video and/or audio annotations 
302. information output to the multimedia streams gen- 
erator includes the multimedia streams 303 in RealAu- 
dio, RealVideo, Real G2 with Flash, RealPix, RealText 
(http://www.realnetworks.com/products/authkit/index. 
html), and ASF file formats 

(http://msdn.microsoft.com/workshop/imedia/windows- 
media/), or any other proprietary file formats, like the 
stream of events defining the graphical annotations. 
Multimedia streams in RealAudio, RealVideo, Real G2 
with Flash, RealPix, RealText file formats can be played 
using RealPlayer. Whereas, the multimedia streams in 
ASF file formats can be played using the Media Player. 
Finally, any other proprietary file format created by the 
multimedia streams generator 250 requires the installa- 
tion of media annotation tool 210 into any client compu- 
ter for playback of the multimedia streams. 
[0023] Referring to FIG 2, there is shown a block dia- 
gram of a preferred embodiment of the present invention 
for generating multimedia presentations 304 and re- 
trieval of these presentations by a client computer 500, 
which includes a web publisher 280, web server 410, 
content server 420, web browser 51 0, and a plugin 520. 
[0024] Amultimediapresentation304isan interactive 
and synchronized audio-visual description of a proce- 
dure, thing, people, or knowledge. It is composed of var- 
ious multimedia streams that can be watched in the cli- 
ent computer screen 550 and listened to by the speak- 
ers 560 attached to the client computer 500. 



[0025] The composition, synchronization and playlist 
of different multimedia streams that compose a multi- 
media presentation are defined but not limited to using 
a scripting language such as the Synchronized Multime- 

s dia Integration Language (SMIL) by RealNetworks 
(http://www.realnetworks.com/products/authkit/index. 
html) or ASX scripting language by Microsoft 
(http://msdn.microsoft.com/workshop/imedia/windows- 
media/CrCont ent/asx.asp). The multimedia presenta- 

10 tion 304 script and a URL link, which is needed to re- 
trieve the multimedia presentation 304 within a web 
browser 510, are generated by the web publisher 280. 
The Web server 410 is responsible for sending the script 
of the multimedia presentation 304 to the client compu- 

15 ter500 over the Internet when requested. On the other 
hand, the content server 420 sends the multimedia 
streams over the Internet to the client computer 500 for 
playback. The plugin 520 decodes the incoming multi- 
media streams into separate buffers and if necessary 

20 creates new buffers, and then renders the buffers to the 
screen 550 or the speakers 560 of the client computer. 
[0026] Referring to FIG 3, there is shown a block dia- 
gram of a preferred embodiment of the present invention 
for generating multimedia messages 305 and retrieval 

25 of these messages by a client computer 500, which in- _ 
eludes an Electronic Mailer 290, electronic mail server 
430, content server 420, Electronic mail (E-mail) client 
530, and the plugin 520. 

[0027] A multimedia message 305 is a multimedia 

30 presentation with recipients' information and optionally 
subject that can be sent as an electronic mail, and 
played back within an E-mail client 530 instead of a web 
browser 51 0. The electronic mail server 430 is respon- 
sible for sending the multimedia message 306 to a list 

35 of recipients. Similarly, the plugin 520 decodes the in- 
coming multimedia presentation corresponding to the 
message into separate buffers and if necessary create 
new buffers and renders the buffers to the screen 550 
or the speakers 560 of the client computer. Similarly, a 

40 URL link, which is generated by the electronic mailer 
290, is needed to access the multimedia message 305. 
The composition, synchronization and playlist of differ- 
ent multimedia streams that compose a multimedia 
message are defined again using a scripting language 

45 such as the SMIL or ASX. The electronic mailer 290 gen- 
erates the script. The content server 420 sends the mul- 
timedia streams over the Internet to the client computer 
500 for playback when a recipient selects to view the 
message. It is desired that the multimedia message start 

50 playing when it is selected from the list of messages in 
the E-mail client 530. This behavior is called one-click 
play. The alternative to one-click play requires perform- 
ing two clicks: (i) opening the multimedia message, and 
(ii) then clicking the URL link within the message body 

55 to start the playback of the multimedia presentation cor- 
responding to the message. One-click play can be im- 
plemented within a Microsoft Outlook Form. 
[0028] Referring to FIG 4, there is shown a block dia- 
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gram of a preferred embodiment of the present invention 
for generating multimedia messages with attachments 
306 and retrieval of these messages by a client compu- 
ter 500, which includes a Electronic Mailer 290, elec- 
tronic mail server 430, E-mail client 530, and the plugin 
520. 

[0029] A multimedia message with attachments 306 
is a message that contains the multimedia streams 
needed for playback of the multimedia presentation cor- 
responding to the message. The multimedia messages 
with attachments are accessed through the E-mail client 
530. The electronic mail server 430 is responsible for 
sending the multimedia message with attachments 306 
to a list of recipients. Similarly, the plugin 520 decodes 
the attached multimedia streams corresponding to the 
presentation into separate buffers and if necessary cre- 
ate new buffers and renders the buffers to the screen 
550 or the speakers 560 of the client computer. There 
is no need for a URL link to access the multimedia pres- 
entation since its components are included in the mes- 
sage. The composition, synchronization and playlist of 
different multimedia streams that compose a multimedia 
message could be defined again using a scripting lan- 
guage such as the SMIL or ASX. The electronic mailer 
290 generates the script. . 

[0030] ' Referring to FIG 5, there is shown a block dia- 
gram of a preferred embodiment of the plugin 520, which 
includes either the media annotation tool 21 0 or stream- 
ing media player 526 depending on the software in- 
stalled on the client computer and its interface with mul- 
timedia presentation 304, multimedia message 305 and 
multimedia message with attachments 306. If the media 
annotation tool 21 0 exists in the client computer 500 (af- 
firmative answer to enable question 522) then the plugin 
520 is the media annotation tool 210. Otherwise (nega- 
tive answerto enable question 522), the plugin is a freely 
available streaming media player 524 like RealPlayer or 
Media Player. The information regarding the existence 
of the media annotation tool 210 can also be stored in 
the multimedia message 305 body. During the multime- 
dia message 305 creation stage, the system can define 
the type of the message and store an integer in the mes- 
sage script for defining the type. The client computer 
500 where the plugin 520 is operating can be one of the 
following three types: (i) with media annotation tool 210; 
(ii) witnout the media annotation tool 210; and (iii) the 
media annotation tool 21 0 with the multimedia message 
305, 306 are received as an attachment to regular E- 
mail. If the media annotation tool 210 with the multime- 
dia message 305, 306 are received at the client compu- 
ter 500 as an attachment then, the this attachment 
launches itself as a self-contained executable and plays 
the multimedia messages 305, 306 automatically when 
it is selected. 

[0031] Referring to FIG. 6, there is shown a block di- 
agram of a preferred embodiment of the media annota- 
tion tool 210, which includes a playback engine 21 2, me- 
dia renderer 214, recording engine 21 6, and mouse and 



keyboard action interpreter 21 8. 
[0032] Information input to the media annotation tool 
21 0 includes the media sources 1 1 0, optionally live vid- 
eo and/or audio capture 120, mouse 123 and keyboard 

s 124 actions in the case of media annotation recording, 
and multimedia presentations 304, multimedia messag- 
es 305, multimedia messages with attachments 306 or, 
hence, multimedia streams 303 in the case of media an- 
notation playback. On the other hand, information out- 

10 put from the media annotation tool 21 0 includes one or 
more of the video and/or audio annotations 302 and 
graphical annotations 301 in the case of media annota- 
tion recording, audio-visual renderings of such annota- 
tions on the screen 125 and the speakers 126 of the 

15 computer in the case of media annotation playback. 
[0033] The playback engine 21 2 is responsible for de- 
composing a multimedia presentation 304, multimedia 
message 305 or multimedia message with attachments 
306 into multimedia streams 303 and then decoding the 

20 streams into buffers or if. necessary creating new buff- 
ers, which are to be rendered to the screen 125 or the 
speakers 126 of the computer by the media renderer 
module 214. The media renderer module 21 4 is also re- 
sponsible for the synchronization, composition and mix- 

25 ing of the data buffers corresponding to different multi- 
media streams, such as compositing graphical annota- 
tions onto images. 

[0034] The mouse and keyboard action interpreter 
21 8 is responsible for converting mouse and keyboard 

30 events into events that represent graphical annotations 
of the input media. Finally, the recording engine 216 
module saves all the incoming information such as 
graphical annotation events and live video and/or cap- 
ture and time stamp each information accordingly to the 

35 their occurrence into annotation files. 

[0035] Referring to FIG. 7, there is shown a block di- 
agram of a preferred embodiment of the multimedia 
streams generator 250, which includes a video genera- 
tion engine 256 and multimedia streams generation en- 

40 gine 258. 

[0036] Information input to the multimedia streams 
generator 250 includes the media sources 110, graphi- 
cal annotations 301 and video and/or audio annotations 
302. Information output from the multimedia streams 

45 generator 250 includes the multimedia streams 303. 
[0037] As described above, It is desirable to create 
multimedia presentations which could be accessed by 
the clients without the need for installing the proprietary 
multimedia annotation tool, which is currently necessary 

so for playing back the graphical annotations, into their 
computers. In the present invention, a stack of images 
with time stamps of the multimedia presentation is gen- 
erated using the slide show presentation generation en- 
gine 256, and then this slide show presentation is con- 

55 verted into a multimedia stream using the multimedia 
stream generation engine 258. The user who is gener- 
ating the multimedia annotations makes the conversion 
decision 252 for generating a slide show presentation. 
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For example, if the user is preparing a multimedia pres- 
entation, which will be posted in the WWW and acces- 
sible by the public by web browsers, then the con version 
decision 252 could most likely be affirmative. If the user 
is preparing a multimedia message 305, 306, then the 5 
multimedia streams generator pops up a dialog window 
on the computer screen for asking the user his/her 
choice. The conversion decision 252 can also be read 
from a database, if the message sender prefers to keep 
a database of recipients' preferences and/orservice lev- 10 
els. There exists a mix decision 254 that has to be taken 
by the multimedia streams generator. The user of the 
system or the system itself may decide to reduce the 
number of multimedia streams 303 that forms a multi- 
media presentation 304 for simplicity. In the case the 15 
video and audio annotations come from different sourc- 
es, the slide show presentation generation engine 256 
can mix the two into a video annotation with mixed 
sound track, since video is nothing but a slide show 
presentation with fixed frame rate. Similarly, the whole 20 
multimedia presentation with graphical, audio and video 
annotations can be converted into a composite slide 
show presentation with mixed sound and visual infor- 
mation. As in the case of the conversion decision 252, 
this mix decision 254 regarding the mixing can be made 25 
by the user using a dialog box or can be read from a 
database.The selection of the images forgenerating the 
image slide show presentation of the multimedia anno- 
tation is an important problem. In the preferred embod- 
iment of this invention, it is suggested to use the screen 30 
dumps of the changing visual data that is displayed with- 
in the media annotation tool at the recording instances 
of the annotations. Since each graphical annotation has 
a time stamp, which screen dump is visible at the time 
of drawing a marker or line is known. The sampling rate 3s 
of the visual data whose screen dumps are captured de- 
pends on the multimedia content being annotated. For 
example for video, the sampling rate is the video sam- 
pling rate, and for 3D animations or any HTML and Of- 
fice document animation, the sampling rate is the rate 40 
of change of the animations specified by the originator 
of such documents. When manipulating a 3D scene or 
an object, or scrolling down or up within the display win- 
dow of the media annotation tool, a screen dump is cre- 
ated at preferably every 40 milliseconds. For each 45 
screen dump, a time stamp is also stored, as is done for 
each graphical annotation. Given the screen dumps and 
thegraphical annotations, aslideshow presentation can 
be generated by overlaying the corresponding drawings 
on top of the screen dumps. so 
[0038] Referring to FIG. 8, there is shown a flow chart 
of a preferred embodiment of the multimedia presenta- 
tion generation by the web publisher 280, which is used 
for getting 282 the name of the content server, generat- 
ing 284 the presentation script that defines the compo- ss 
sition, interactions and synchronization of the multime- 
dia presentation 304 of the multimedia streams, script 
uploading 286 the presentation script to a web server, 



and finally stream uploading 288 the multimedia 
streams to the content server. As described earlier, the 
scripting languages such as SMIL and ASX can be used 
to create the presentation script. 
[0039] Referring to FIG. 9, there is shown a flow chart 
of a preferred embodiment of the multimedia message 
305 or multimedia message with attachments 306 gen- 
eration by the electronic mailer 290. 
[0040] The initial step of the message generation is to 
decide, through content server decision 291 , what type 
of message to send. If the computer system where the 
electronic mailer is operating has a content server then 
the content server decision 291 is affirmative and the 
following steps are carried out to send the multimedia 
message 305: (i) getting the name of the content server 
292; (ii) generating the presentation script that defines 
the composition, interactions and synchronization of the 
multimedia presentation of the multimedia streams 293; 

(iii) generating the e-mail message by defining its type 
and generating the URL link that points to the multime- 
dia presentation 294; (iv) uploading the multimedia 
streams.to the content server 295; and (v) finally getting 
the recipients' addresses from the user and then send- 
ing the multimedia message 296. If there does not exist 
a content server (negative decision at content server de- 
cision 291 ), then the following steps are carried out for 
sending the multimedia message: (i) generating the 
presentation script that defines the composition, inter- 
actions and synchronization of the multimedia presen- 
tation of the multimedia streams 293; (ii) generating the 
e-mail message by defining its type 294; (iii) attaching 
the multimedia streams to the e-mail message 297; and 

(iv) finally getting the recipients' addresses from the user 
and then sending the multimedia message with attach- 
ments 296. 

[0041 ] It is not intended that the present invention be 
limited to the hardware or software arrangement, or op- 
erational procedures shown disclosed. This invention 
includes all of the alterations and variations thereto as 
encompassed within the scope of the claims as follows. 



Claims 

1 . A system for generating and playback of annotated 
multimedia presentation comprising: 

a media annotation tool having both recording 
and playback abilities; 

a multimedia streams generator comprising a 
video generation engine; 
a web publisher; and 
an electronic mailer. 

2. The system as recited in claim 1 , wherein the elec- 
tronic mailer is adapted to send an annotated mul- 
timedia presentation via an electronic mail mes- 
sage, wherein said media annotation tool is sent as 
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a self-contained executable attachment to said 
electronic mail message. 

The system as recited in claim 1 , adapted to retrieve 
an annotated multimedia presentation sent via an s 
electronic mail message by means of a URL em- 
bedded in said electronic mail message, wherein 
said annotated multimedia presentation is 
streamed to said electronic mailerth rough said em- 
bedded URL. 10 

The system as recited in claim 3, being adapted to 
read an integer stored at the beginning of the anno- 
tated multimedia presentation, to enable the deter- 
mination of the type of annotated multimedia pres- f 5 
entation. 
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